Arrangement, method and computer program for controlling a computer apparatus based on eye-tracking

ABSTRACT

A computer based eye-tracking solution is disclosed. A computer apparatus is associated with one or more graphical displays (GUI components) that may be manipulated based on user-generated commands. An event engine is adapted to receive an eye-tracking data signal that describes a user&#39;s point of regard on the display. Based on the signal, the event engine is adapted to produce a set of non-cursor controlling event output signals, which influence the GUI-components. Each non-cursor controlling event output signal describes a particular aspect of the user&#39;s ocular activity in respect of the display. Initially, the proposed event engine receives a control signal request from each of the GUI-components. The control signal request defines a sub-set of the set of non-cursor controlling event output signals which is required by the particular GUI-component. The event engine delivers non-cursor controlling event output signals to the GUI-components in accordance with each respective control signal request.

THE BACKGROUND OF THE INVENTION AND PRIOR ART

The present invention relates generally to computer based eye-trackingsystems. More particularly the invention relates to an arrangement forcontrolling a computer apparatus according to the preamble of claim 1and a corresponding method according to the preamble of claim 11. Theinvention also relates to a computer program according to claim 21 and acomputer read-able medium according to claim 22.

The human computer interaction was revolutionized by the introduction ofthe graphical user interface (GUI). Namely, thereby, an efficient meanswas provided for presenting information to a user with a bandwidth whichimmensely exceeded any prior channels. Over the years the speed at whichinformation can be presented has increased further through colorscreens, enlarged displays, intelligent graphical objects (e.g. pop-upwindows), window tabs, menus, toolbars and sounds. During this time,however, the input devices have remained essentially unchanged, i.e. thekeyboard and the pointing device (e.g. mouse, track ball or touch pad).In recent years, handwriting devices have been introduced (e.g. in theform of a stylus or graphical pen). Nevertheless, while the outputbandwidth has multiplied several times, the input ditto has beensubstantially unaltered. Consequently, a severe asymmetry in thecommunication band-width in the human computer interaction has occurred.

In order to decrease this bandwidth gap, various attempts have been madeto use eye-tracking devices. However, in many cases these devices missthe mark in one or several respects. One problem is that the prior-artsolutions fail to take a holistic view on the input interfaces to thecomputer. Thereby, comparatively heavy motor tasks may be imposed on theeyes, which in fact are strictly perceptive organs. Typically, thisleads to fatigue symptoms and a degree of discomfort experienced by theuser. This is particularly true if an eye tracker is used to control acursor on a graphical display, and for various reasons the eye trackerfails to track the user's point of regard sufficiently well, so thatthere is a mismatch between the user's actual point of regard and theposition against which the cursor is controlled.

Instead of controlling the cursor directly, an eye gaze signal may beused to select an appropriate initial cursor position. The document U.S.Pat. No. 6,204,828 discloses an integrated gaze/manual cursorpositioning system, which aids an operator to position a cursor byintegrating an eye-gaze signal and a manual input. When a mechanicalactivation of an operator device is detected the cursor is placed at aninitial position which is predetermined with respect to the operator'scurrent gaze area. Thus, a user-friendly cursor function isaccomplished.

The document U.S. Pat. No. 6,401,050 describes a visual interactionsystem for a shipboard watch station. Here, an eye-tracking cameramonitors an operator's visual scan, gaze location, dwell time, blinkrate and pupil size to determine whether additional cueing of theoperator should be made to direct the operator's attention to animportant object on the screen.

The document U.S. Pat. No. 5,649,061 discloses a device for estimating amental decision to select a visual cue from a viewer's eye fixation andcorresponding event evoked cerebral potential. An eye tracker registersa viewing direction, and based thereon fixation properties may bedetermined in terms of duration, start and end pupil sizes, saccades andblinks. A corresponding single event evoked cerebral potential isextracted, and an artificial neural network estimates a selectioninterest in the gaze point of regard. After training the artificialneural network, the device may then be used to control a computer, suchthat icons on a display are activated according to a user's estimatedintentions without requiring any manipulation by means of the user'shands.

A few attempts have also been made to abstract user-generated input datainto high-level information for controlling a computer. For example, thedocument US 2004/0001100 describes a multimode user interface, where aflexible processing of a user input is made possible without having toswitch manually between different input modes. Instead, within the datastreams different information categories are distinguished depending ona context in which the data streams are generated.

Although this strategy may indeed enhance the efficiency of theman-machine interaction, no multimodal solution has yet been presentedaccording to which eye-tracking data is processed optimally. On thecontrary, with only very few exceptions, today's eye tracking interfacesare each tailored for one specific task only. Thus, any processing ofeye tracking data in respect of a first application cannot be reused bya second application, and vice versa. Hence, if multiple eye-controlledapplications are used in a single computer, one particular eye-trackingdata processing unit is typically required for each application.Naturally, in such a case, there is a high risk that the differentapplications perform a substantial amount of overlapping eye-trackingdata processing. Moreover, each designer of eye-controllableapplications needs to have expertise in both eye tracking technology andthe interpretation of eye-tracking data in order to extract the datarequired by the application.

SUMMARY OF THE INVENTION

The object of the present invention is therefore to provide a holisticmeans of controlling a computer apparatus based on a user's ocularactivity, which alleviates the above problems and thus offers anefficient man-machine interaction with a minimal amount of doubleprocessing, i.e. wherein as much as possible of the eye-tracking dataprocessing is performed centrally in respect of GUI-components that maybelong to two or more separate applications.

According to one aspect of the invention, the object is achieved by thearrangement as initially described, wherein the event engine is adaptedto receive a control signal request from each of the at least oneGUI-component. The control signal request defines a sub-set of the setof non-cursor controlling event output signals, which is required by theGUI-component in question. The event engine is also adapted to delivernon-cursor controlling event output signals to the at least oneGUI-component in accordance with each respective control signal request.

This arrangement is advantageous because thereby a very flexibleinterface is attained towards any applications which are controllable bymeans of eye tracking signals. This, in turn, enables softwaredevelopers without an in-depth knowledge of eye-tracking technology todesign eye-controllable applications. Therefore, the invention isbelieved to stimulate the development of new such applications, andconsequently render further improvements of the human computerinteraction possible. Moreover, the computer's processing resources arefreed for alternative purposes, since a high-level eye-tracking datasignal derived in respect of one application may be reused by one ormore additional applications. In other words, the event engine isadapted to perform a centralized eye-tracking data processing forGUI-components (or potential “subscribers”) on-demand. Thus, each ofthese components and their requested control signals need not beexplicitly known at the point in time when the event engine isimplemented. Instead, according to the invention, one or moreGUI-components may request relevant control signals in connection withthe implementation of a later added eye-controllable application. Ofcourse, the opposite may also be true, namely that a set of controlsignal requests is presented once and for all at start-up of theproposed arrangement.

According to a preferred embodiment of this aspect of the invention, thecomputer apparatus is also adapted to receive a cursor control signal,and in response to the cursor control signal, control a graphicalpointer on the display. Thus, for instance, the event engine and thecursor control signal may interact jointly with GUI-componentsrepresented on the display, such that a very intuitive man-machineinteraction is accomplished.

According to another preferred embodiment of this aspect of theinvention, at least one GUI-component is adapted to generate at leastone respective output control signal upon a user manipulation of thecomponent. This means that by manipulating a GUI-component, the user maycause the computer apparatus to generate outgoing signals to one or moreexternal units, such as a printer, a camera etc. Naturally, this is ahighly desirable feature. Moreover, the fact that one or moreGUI-components may generate an output control signal does not precludethat with respect to one or more other GUI-components, the non-cursorcontrolling event output signals may exclusively affect the componentinternally.

According to a preferred embodiment of this aspect of the invention, theevent engine is adapted to produce at least a first signal of thenon-cursor controlling event output signals based on a dynamicdevelopment of the eye-tracking data signal. Thereby, a time parameterof the user's ocular activity may be used to control functions andprocesses of the computer apparatus. For instance, the time parametermay reflect a dwell time of the user's gaze within a particular regionon the display, identify a certain gaze pattern etc. Many types ofadvanced eye-controllable functions can thereby be realized.

According to yet another preferred embodiment of this aspect of theinvention, at least one GUI-component is adapted to interpret anon-cursor controlling event output signal as a representation of theuser's intention. In response to this estimated intention, a usermanipulation of the component is triggered. For example this may involveactivating one or more computer functions based on a command history.This is advantageous because thereby the command input procedure may besimplified.

According to still another preferred embodiment of this aspect of theinvention, at least one GUI-component is adapted to interpret anon-cursor controlling event output signal as an estimated attentionlevel of the user. In response to this estimated attention level, a usermanipulation of the component is triggered. One advantage attainablethereby is that the computer's behavior may be adapted to match acurrent attention level of the user.

According to yet another preferred embodiment of this aspect of theinvention, at least one GUI-component is adapted to interpret anon-cursor controlling event signal as a state-of-mind parameter of theuser, and in response thereto trigger a user manipulation of thecomponent. This feature is desirable because it allows the computer tobehave differently depending on whether the user appears to befocused/concentrated, distracted, tired/unfocused or confused etc.

According to still another preferred embodiment of this aspect of theinvention, the event engine is adapted to receive at least one auxiliaryinput signal, such as a signal from a button or a switch, a speechsignal, a movement pattern of an input member, a camera registeredgesture pattern or facial expression, or an EEG(electroencephalogram)-signal. On the further basis of this auxiliarysignal the event engine produces the set of non-cursor controlling eventoutput signals. Thereby, a highly efficient combination of user inputsignals may be used to control the computer. Generally, thesecombinations can be made very intuitive and easy to learn.

According to another aspect of the invention the object is achieved bythe method as initially described, wherein a control signal request isreceived from each of the at least one GUI-component. The control signalrequest defines a sub-set of the set of non-cursor controlling eventoutput signals, which is required by the particular GUI-component.Non-cursor controlling event output signals are then delivered to the atleast one GUI-component in accordance with each respective controlsignal request.

The advantages of this method, as well as the preferred embodimentsthereof, are apparent from the discussion hereinabove with reference tothe proposed arrangement.

According to a further aspect of the invention the object is achieved bya computer program, which is directly loadable into the internal memoryof a computer, and includes software for controlling the above proposedmethod when said program is run on a computer.

According to another aspect of the invention the object is achieved by acomputer readable medium, having a program recorded thereon, where theprogram is to control a computer to perform the above proposed method.

The invention dramatically increases the available bandwidth fortransferring information from a user to a computer apparatus, i.e.essentially generating commands, however not necessarily perceived assuch by the user. Therefore, this increase of the bandwidth places noadditional cognitive workload on the user. On the contrary, by means ofthe invention, the cognitive workload may, in fact, be reduced. At thesame time, the increased bandwidth vouches for an improved efficiency ofthe man-machine interaction.

Moreover, by means of the invention, commands which traditionally haverequired hand and/or finger manipulations, can be efficiently andeffortlessly effected based on the user's eye activity. Naturally, thisis desirable in a broad range of applications, from disabled computerusers, support operators in a call-center environment (e.g. whenentering/editing data in a customer relationship managementapplication), users of advanced computer aided design (CAD) tools,surgeons, to drivers and pilots who, for various reasons, cannoteffectively produce hand- and/or finger-based commands. Even insituations where the users have their hands and fingers readilyaccessible, the invention may be useful to improve the ergonomics andthe reduce risk of e.g. repetitive strain injuries. Alternatively, theenvironment in which the computer apparatus is placed may be so clean ordirty that either the environment has to be protected from possibleemissions from the computer apparatus, or reverse this apparatus must beprotected against hazardous substances in the environment and thereforehas to be encapsulated to such a degree that a traditional entry ofinput of commands is made impossible, or at least impractical.

The invention offers an excellent basis for developing new software andcomputer applications, which are controllable by the eyes of a user.Thereby, in the long term, the invention vouches for a deep andunconstrained integration of eye interaction applications into thestandard computer environment.

Further advantages, advantageous features and applications of thepresent invention will be apparent from the following description andthe dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is now to be explained more closely by means ofpreferred embodiments, which are disclosed as examples, and withreference to the attached drawings.

FIG. 1 shows an overview picture of a user controlling a computerapparatus according to the invention;

FIG. 2 shows an arrangement for controlling a computer apparatusaccording to an embodiment of the invention;

FIGS. 3 a-b show proposed symbols representing an eye-controllableGUI-component on a display in a non-observed and an observed moderespectively;

FIG. 4 illustrates a first embodiment according to the invention, wherea proposed multiview toolbar is used;

FIG. 5 illustrates a second embodiment according to the invention basedon the proposed multiview toolbar;

FIGS. 6 a-b illustrate a third embodiment according to the invention,where screen controls are adapted to expand upon a manipulation, whichis based on a user's ocular activity;

FIG. 7 illustrates a fourth embodiment according to the invention, whichrealizes a scrolling function based on a user's ocular activity; andFIG. 8 illustrates, by means of a flow diagram, a general method ofcontrolling a computer apparatus according to the invention.

DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

FIG. 1 shows an overview picture of a typical use-case according to theinvention. Here, a user 110 controls a computer apparatus 130, at leastpartly based on an eye-tracking data signal D_(EYE), which describes theuser's 110 point of regard x, y on a display 120. Thus, by watching arepresentation of a GUI-component 220 on the display 120, the user 110may generate commands to the computer apparatus 130. This manipulationis enabled, since the GUI-component 220 is adapted to be, at leastindirectly, influenced by the eye-tracking data signal D_(EYE). Theinvention presumes that the eye-tracking data signal D_(EYE) may resultin events, related to any task performable by the computer apparatus,apart from affecting a cursor/pointer on the display 120. It should benoted that according to the invention, any type of known computer screenor monitor, as well as combinations of two or more separate displays mayrepresent the display 120. For example, the display 120 may constitute apair of stereoscopic screens, a heads-up display (HUD), a head-mounteddisplay (HMD) and a presentation means for a virtual environment, suchas the eyepieces of a pair of 3D-glasses or a room where the wallsinclude projection screens for presenting a virtual environment.

Naturally, in order to produce the eye-tracking data signal D_(EYE), thedisplay 120 is associated with, or includes, an eye-tracker. This unitis not a subject of the present patent application, and therefore willnot be described in detail here. However, the eye-tracker is preferablyembodied by the solution described in the Swedish patent application0203457-7, filed on 21 Nov. 2002 in the name of the applicant.

Preferably, a graphics control signal C-GR is generated by the computerapparatus 130 to accomplish visual feedback information on the display120. The visual feedback information is generated in response to anyuser-commands received by the computer apparatus 130, so as to confirmto the user 110 any commands, which are based on the eye-tracking datasignal D_(EYE). Such a confirmation is especially desirable when thecommands are produced by means of perceptive organs, for example thehuman eyes.

According to a preferred embodiment of the invention, besides theeye-tracking data signal D_(EYE), the computer apparatus 130 is adaptedto receive a cursor control signal K, which controls the position of agraphical pointer on the display 120. Of course, the graphics controlsignal C-GR may be based also on the cursor control signal K.

FIG. 2 shows an arrangement according to an embodiment of the invention,which may be realized by means of the computer apparatus 130 describedabove with reference to the FIG. 1.

The arrangement includes an event engine 210 and at least oneGUI-component 220, which is adapted to be manipulated based onuser-generated commands, at least partly expressed by the eye-trackingdata signal D_(EYE). The event engine 210 is adapted to receive theeye-tracking data signal D_(EYE), and based thereon produce a set ofnon-cursor controlling event output signals D-HI_(i) that influence theat least one GUI-component 220. Each non-cursor controlling event outputsignal D-HI_(i), in turn, describes a particular aspect of the user's110 ocular activity in respect of the display 120.

For example, a first signal may indicate whether the user's 110 gaze isdirected towards the display at all (i.e. a “gaze-on-display” signal), asecond non-cursor controlling event output signal may reflect a dwelltime of the user's 110 gaze within a, certain area on the display 120, athird signal may designate a gaze fixation (at a specific point), afourth signal may indicate whether the gaze saccades, a fifth signal mayindicate whether the gaze follows a smooth path, a sixth signal mayreflect that the user 110 reads a text, and a seventh signal may betriggered if the user 110 appears to be distracted, based on theparticular eye-tracking data signal D_(EYE) that he/she produces.

According to the invention, the event engine 210 receives a respectivecontrol signal request R_(a), . . . , R_(n) from each of the at leastone GUI-component 220 a, . . . , 220 n. The control signal request, sayR_(a), defines a sub-set of the set of non-cursor controlling eventoutput signals D-HI_(i) which are required by the particularGUI-component, say 220 a, to operate as intended. The event engine 210then delivers non-cursor controlling event output signals D-HI_(i) toeach of the at least one GUI-component 220 a, . . . , 220 n inaccordance with the respective control signal request R_(a), . . . ,R_(n).

A most efficient processing is accomplished if the event engine 210exclusively produces those event output signals D-HI_(i) which areactually requested by at least one GUI-component. However, according tothe invention, it is also conceivable that all non-cursor controllingevent output signals D-HI_(i) that are possible to produce are alwaysgenerated by the event engine 210, irrespective of whether acorresponding control signal request has been received or not. Namely,this simplifies the procedure, and depending on the application, thisstrategy may not require an overly extensive processing.

According to a preferred embodiment of the invention, each GUI-component220 a, . . . , 220 n is adapted to generate at least one respectiveoutput control signal C_(a), . . . , C_(a) upon a user manipulation ofthe component 220 a, . . . , 220 n. Thus, in addition to generating theabove-mentioned visual feedback information, one or more internal orperipheral devices may be influenced by means of the output controlsignals C_(a), . . . , C_(a). For example a print job may be initiated,a computing task may be executed, an e-mail may be sent out, a cameramay be triggered to take a picture etc.

As mentioned above, the non-cursor controlling event output signalsD-HI_(i) may describe many different aspects of the eye-tracking datasignal D_(EYE). According to one embodiment of the invention, at leastone output signal D-HI_(i) is based on a dynamic development of theeye-tracking data signal D_(EYE). Thereby, the signal can represent aparticular gaze pattern over the display 120. The gaze pattern, in turn,may be determined, for instance to constitute saccades, a smoothpursuit, periods of fixation or reading.

A non-cursor controlling event output signal D-HI_(i) may also indicategaze-enter/gaze-leave data. This data is a parameter which reflects thetime instances when the eye-tracking data signal D_(EYE) indicates thatthe user's point of regard falls within a GUI component's representationon the display. Hence, the gaze-enter data is generated when the user'sgaze falls onto a GUI component's representation on the display, and thegaze-leave data is generated when the gaze is directed outside thisrepresentation.

The above-mentioned dwell time for a GUI-component is typically definedas the period between the gaze-enter data and the gaze-leave data withrespect to a particular GUI component. It is normally preferable to linkan activation signal to the dwell time, such that for instance an“eye-button” is activated when a certain dwell time is reached.Alternatively, a button, a switch, a speech signal, a movement patternof an input member, a camera registered gesture pattern or facialexpression may constitute the activation signal. Moreover, an eye blinkor a predefined EEG-signal may cause the activation signal. However, thelatter types of signals are usually relatively difficult for the user tocontrol with a sufficient accuracy.

According to one embodiment of the invention, the event engine 210 isadapted to receive at least one auxiliary input signal D_(J), andproduce the set of non-cursor controlling event output signals D-HI_(i)on the further basis of this signal. The auxiliary input signal D_(J)may originate from a button, a switch, a speech signal, a movementpattern of an input member, a camera registered gesture pattern or afacial expression, or an EEG-signal.

Thereby, based on combinations of the eye-tracking data signal D_(EYE)and one or more auxiliary input signals D_(J) composite user commandsmay be created, which are very intuitive and easy to learn. Hence, ahighly efficient man-machine interaction with the computer apparatus 130may be accomplished. For instance, watching an eye-button for thedocument and uttering the control word “open” can open a text document.If a more complex speech recognition is available, an Internet searchmay be effected by focusing the gaze towards a relevant text input boxwhile uttering the desired search terms, and so on.

According to one embodiment of the invention, at least one GUI-component220 a, . . . , 220 n is adapted to interpret a non-cursor controllingevent output signal D-HI_(i) from the event engine 210 as an estimatedintention of the user 110. Then, in response to the estimated intention,a user manipulation of the component 220 a, . . . , 220 n is triggered.Preferably, the event engine 210 estimates a user intention based onmultiple input sources received as auxiliary input signals D_(J), whichmay include a key-board signal, a mouse signal, voice data and cameraimages. However, naturally, the eye-tracking data signal D_(EYE) mayalso constitute a basis for the estimated user intention. For example,important information can be drawn from different gaze patterns andfixation times.

According to another embodiment of the invention, at least oneGUI-component 220 a, . . . , 220 n is adapted to interpret a non-cursorcontrolling event output signal D-HI_(i) from the event engine 210 as anestimated attention level of the user 110. Correspondingly, a usermanipulation of the component 220 a, . . . , 220 n is triggered inresponse to the estimated attention level. Also the attention level maybe estimated based on the auxiliary input signals D_(J), for instanceoriginating from the keyboard, the mouse, voice data and camera images,and the eye-tracking data signal D_(EYE). Particularly, gaze patterns,fixation points and fixation times constitute an important basis fordetermining the user's 110 attention level. Preferably, theGUI-components 220 a, . . . , 220 n vary their behavior depending on theestimated attention level, such that the components' characteristicsmatch the user's 110 current performance.

Furthermore, according to one embodiment of the invention, at least oneGUI-component 220 a, . . . , 220 n is adapted to interpret a non-cursorcontrolling event output signal D-HI_(i) from the event engine 210 as astate-of-mind parameter of the user 110. The state-of-mind parameterreflects a general user 110 status, for example whether he/she appearsto be focused/concentrated, distracted, tired/unfocused or confused. Forexample, the state-of-mind parameter may indicate an approximated 20%degree of tiredness and an approximated 50% degree of attention. Then,based on the estimated state-of-mind, a user manipulation of thecomponent 220 a, . . . , 220 n is triggered. Typically, the number ofand the contents of any help menus and pop-up windows may be adapted inresponse to the estimated state-of-mind. How-ever, in operatorenvironments, such as radar watch stations where the attention level aswell as the state-of-mind may be truly critical, the security may beimproved by pointing out targets etc. that the operator has not yetobserved.

According to one embodiment of the invention, the event engine 210 isassociated with a template library 230, which contains genericGUI-components, such as eye-buttons, scroll bars, multiview toolbars(see below with reference to FIG. 4), text input fields (see below withreference to FIG. 5), expandable text input fields (see below withreference to FIG. 6) and scroll windows (see below with reference toFIG. 7). Thereby, by means of the template library 230, a softwaredesigner may conveniently create functions and controls which can bemanipulated based on a user's ocular activity in respect of a display.Of course, after completing the design of a particular eye-controllableapplication, however, the template library 230 has no actual function inrespect of the GUI-components that may originate from its genericcomponents. Nevertheless, the template library 230 may again becomeuseful in case of a future upgrade or a redesign of the application.

According to the invention, it is not technically necessary that anyGUI-component manipulations be graphically confirmed on the display.However, this is generally preferable from a user-friendlinesspoint-of-view. FIG. 3 a shows a schematic symbol 310, which representsan eye-controllable GUI-component on a display that is set in anon-observed mode (i.e. the eye-tracking data signal D_(EYE) indicatesthat the user's point of regard lies outside the symbol 310). FIG. 3 bshows the symbol 310 in an observed mode, which is set when theeye-tracking data signal D_(EYE) indicates that the user's point ofregard falls within the display area represented by the symbol 310. Inthe observed mode, the symbol 310 contains a centrally located object311. This object 311 confirms to the user that the computer apparatushas registered that his/her gaze presently is directed towards thesymbol 310. Thus, any manipulations in respect of the GUI-componentassociated with the symbol 310 can be performed. An important advantageattained by means of the centrally located object 311 is that thisobject assists the user in focusing his/her gaze to the center of thesymbol 310. Thereby, a more reliable eye-tracking function isaccomplished, and for a given eye-tracker, the symbols 310 can be madesmaller than otherwise. Of course, the symbol 310 and the centrallylocated object 311 may have any other outline than the squarerepresentation of FIGS. 3 a and 3 b. Moreover, to further improve thevisual cueing, the object 311 may be animated and or have a particularlyinteresting color or shape.

FIG. 4 illustrates a first embodiment according to the invention, wherea proposed multiview toolbar 401 is used to control applications in aframe 400. The multiview toolbar 401 here includes four differenteye-buttons, which each may contain a thumbnail image (not shown) of arespective application with which the button is associated. A firstbutton 410 is associated with a first application, e.g. an Internetbrowser, which preferably accesses a predefined URL or web page. Thefirst application has a user interface which here is represented by asub-frame 420 within the frame 400. Hence, by viewing the first button410, the user may open the sub-frame 420. Either this activation isaccomplished after a particular gaze dwell time in respect of the button410, or in response to an activation signal, for example a keystroke ora control word. Then, a search may be executed by viewing a text inputbox 411, entering relevant search terms, and thereafter manipulating asearch button 412 (preferably also based on the eye-tracking datasignal).

FIG. 5 illustrates a second embodiment according to the invention basedon the proposed multiview toolbar 401. Here, a second button 510 isassociated with a second application, e.g. a product management system,which has a user interface in the form of a sub-frame 520 within theframe 400. This means that a user may activate the second application byviewing the second button 510 during a particular gaze dwell time, orgenerating a separate activation signal (as mentioned above).Preferably, a default set of text input boxes and buttons 515 areactivated initially. Then, by viewing other areas within the sub-frame520, alternative fields and functions may be activated and manipulated.

Particularly, a screen control may be adapted to expand upon amanipulation based on a user's ocular activity. FIGS. 6 a and 6 billustrate this as a third embodiment according to the invention.

Here, a text field 620 in the sub-frame 520 occupies a relatively smallarea on the display as long as the eye-tracking data signal indicatesthat the user's point of regard lies outside this field 620. However if,according to the eye-tracking data signal, the point of regard reaches aparticular dwell time within the display area represented by the textfield 620, this field expands, for instance as shown in the FIG. 6 b.Hence, more information than what was initially visible in the field 620may be shown. The thus expanded text field 620 may even cover graphicalobjects which otherwise are shown in the frame 400. This isadvantageous, because thereby the information may be presentedon-demand, so that the frame 400 and the sub-frame 520 contain more datathan what actually can be fitted therein. For instance, a textscrolling, which otherwise would have been necessary can be avoided.

FIG. 7 illustrates a fourth embodiment according to the invention, whichrealizes a scrolling function based on a user's ocular activity. Here, athird application, e.g. a map viewer, is associated with a third button710. Thus, by watching the third button 710, and activating itsassociated GUI-component, either based on a gaze dwell time or by meansof a separate activation signal, the computer apparatus opens up a mapsub-frame 720 within the frame 400. This sub-frame, in turn, presents adigitized map. It is here presumed that the map is larger than theavailable presentation area in the map sub-frame 720, so that only apart of the map can be presented at the same time.

A scrolling with respect to the map is achieved based on the user'spoint or regard. Preferably, no scrolling occurs as long as theeye-tracking data signal indicates that the point of regard lies withina central area delimited by a first dashed line a, a second dashed lineb, a third dashed line c and a fourth dashed line d, neither of whichpreferably are visible on the display. If however, the user's point ofregard is placed outside any of the lines a, b, c or d, the map scrollsin a particular direction given by the point of regard. Specifically,this means that a point of regard below the line c results in a downwardscroll along the arrow S; a point of regard above the line d results ina upward scroll along the arrow N; a point of regard to the right of theline b results in a rightward scroll along the arrow E; and a point ofregard to the left of the line a results in a leftward scroll along thearrow W. Moreover, a point of regard, which lies below the line c and tothe right of the line b results in a diagonal scroll along the arrow SE;a point of regard, which lies above the line d and to the right of theline b results in a diagonal scroll along the arrow NE; a point ofregard, which lies above the line d and to the left of the line aresults in a diagonal scroll along the arrow NW; and a point of regard,which lies below the line c and to the left of the line a results in adiagonal scroll along the arrow SW. This scroll function may either beactivated based on a dwell time, or a separate activation signal may berequired, such as a clicking a key/button or holding down a key/button.

Furthermore, the scroll speed may depend on the distance between thepoint of regard and the respective lines a, b, c and d, such that arelatively long distance corresponds to a comparatively high speed, andvice versa. The scroll speed may also depend on the scroll time, and/ora length of the latest saccades registered in the eye-tracking datasignal. Preferably, a maximum scroll speed is set to such a value thatthe scrolled information is visible to the user at all possible speeds.

According to one embodiment of the invention, the scroll function isstopped by pressing a key/button, releasing a key/button, the length ofthe latest saccades exceeding a particular value, the point of regardmoving outside the map sub-frame 720, the point of regard moving towardsthe center of the map sub-frame 720, or towards an opposite scrollactivation line a, b, c, or d.

It should be noted that, eyetrack-driven scrolling solutions as such aredescribed in the prior art, for instance in the document U.S. Pat. No.5,850,221. Here, a page oriented or continuous information scrollingfunction is initiated or controlled based on where a viewer's eyes arelooking.

To sum up, the general method of controlling a computer apparatusaccording to the invention will now be described with reference to theflow diagram in FIG. 8.

An initial step 810 receives a control signal request from each of atleast one GUI-component, which is adapted to be manipulated based onuser-generated eye commands. The control signal request defines asub-set of the set of non-cursor controlling event output signals, whichis required by the particular GUI-component in order to operate asintended.

A step 820 then receives an eye-tracking data signal, which describes auser's point of regard on a display associated with a computer thatembodies the at least one GUI-component. Subsequently, a step 830produces a set of non-cursor controlling event output signals based onthe eye-tracking data signal, plus any auxiliary input signal.

Subsequently, a step 840 delivers the non-cursor controlling eventoutput signals to the at least one GUI-component so that a relevantinfluence thereof may occur (i.e. according to each respective controlsignal request). It is presumed that each non-cursor controlling eventoutput signal describes a particular aspect of the user's ocularactivity in respect of the display. Thus, the non-cursor controllingevent output signals express user-generated eye commands. After the step840, the procedure loops back to the step 820 for receiving an updatedeye tracking data signal.

All of the process steps, as well as any sub-sequence of steps,described with reference to the FIG. 8 above may be controlled by meansof a programmed computer apparatus. Moreover, although the embodimentsof the invention described above with reference to the drawings comprisecomputer apparatus and processes performed in computer apparatus, theinvention thus also extends to computer programs, particularly computerprograms on or in a carrier, adapted for putting the invention intopractice. The program may be in the form of source code, object code, acode intermediate source and object code such as in partially compiledform, or in any other form suitable for use in the implementation of theprocess according to the invention. The program may either be a part ofan operating system, or be a separate application. The carrier may beany entity or device capable of carrying the program. For example, thecarrier may comprise a storage medium, such as a Flash memory, a ROM(Read Only Memory), for example a CD (Compact Disc) or a semiconductorROM, an EPROM (Erasable Programmable Read-Only Memory), an EEPROM(Electrically Erasable Programmable Read-Only Memory), or a magneticrecording medium, for example a floppy disc or hard disc. Further, thecarrier may be a transmissible carrier such as an electrical or opticalsignal which may be conveyed via electrical or optical cable or by radioor by other means. When the program is embodied in a signal which may beconveyed directly by a cable or other device or means, the carrier maybe constituted by such cable or device or means. Alternatively, thecarrier may be an integrated circuit in which the program is embedded,the integrated circuit being adapted for performing, or for use in theperformance of, the relevant processes.

The term “comprises/comprising” when used in this specification is takento specify the presence of stated features, integers, steps orcomponents. However, the term does not preclude the presence or additionof one or more additional features, integers, steps or components orgroups thereof.

The invention is not restricted to the described embodiments in thefigures, but may be varied freely within the scope of the claims.

1. An arrangement for controlling a computer apparatus associated with agraphical display, the display presenting a representation of at leastone GUI-component which is adapted to be manipulated based onuser-generated commands, the arrangement comprising an event engineadapted to receive an eye-tracking data signal describing a user's pointof regard on the displays, and at least based on the eye-tracking datasignal produce a set of non-cursor controlling event output signalsinfluencing the at least one GUI-component, each non-cursor controllingevent output signal describing a particular aspect of the user's ocularactivity in respect of the display, characterized in that the eventengine is adapted to: receive a control signal request from each of theat least one GUI-component, the control signal request defining asub-set of the set of non-cursor controlling event output signals whichis required by the GUI-component, and deliver non-cursor controllingevent output signals to the at least one GUI-component in accordancewith each respective control signal request.
 2. An arrangement accordingto claim 1, wherein the computer apparatus is adapted to: receive acursor control signal, and control a graphical pointer on the display inresponse to the cursor control signal.
 3. An arrangement according toclaim 1, wherein at least one of the at least one GUI-component isadapted to generate at least one respective output control signal upon auser manipulation of the component.
 4. An arrangement according to claim1, wherein the event engine is adapted to produce at least a firstsignal of the non-cursor controlling event output signals based on adynamic development of the eye-tracking data signal.
 5. An arrangementaccording to claim 4, wherein the first signal represents a particulargaze pattern over the display.
 6. An arrangement according to claim 5,wherein the at least one GUI-component is adapted to interpret the firstsignal as an estimated intention of the user, and trigger a usermanipulation of the component in response to the estimated intention. 7.An arrangement according to claim 5, wherein the at least oneGUI-component is adapted to interpret the first signal as an estimatedattention level of the user, and trigger a user manipulation of thecomponent in response to the estimated attention level.
 8. Anarrangement according to claim 5, wherein the at least one GUI-componentis adapted to interpret the first signal as a state-of-mind parameter ofthe user, and trigger a user manipulation of the component in responseto the state-of-mind parameter.
 9. An arrangement according to claim 1,wherein the event engine is adapted to receive at least one auxiliaryinput signal and produce the set of non-cursor controlling event outputsignals on the further basis of the at least one auxiliary input signal.10. An arrangement according to claim 1, wherein the at least oneauxiliary input signal is originated from at least one of a button, aswitch, a speech signal, a movement pattern of an input member, agesture pattern, a facial expression and an EEG-signal.
 11. A method ofcontrolling a computer apparatus associated with graphical display, thedisplay representing at least one GUI-component which is adapted to bemanipulated based on user-generated commands, the method comprising:receiving an eye-tracking data signal which describes a user's point ofregard on the display, producing a set of non-cursor controlling eventoutput signals at least based on the eye-tracking data signal, the setof non-cursor controlling event output signals influencing the at leastone GUI-component, and each non-cursor controlling event output signaldescribing a particular aspect of the user's ocular activity in respectof the display, receiving a control signal request from each of the atleast one GUI-component, the control signal request defining a sub-setof the set of non-cursor controlling event output signals which isrequired by the GUI-component, and delivering non-cursor controllingevent output signals to the at least one GUI-component in accordancewith each respective control signal request.
 12. A method according toclaim 11 further comprising receiving a cursor control signal, andcontrolling a graphical pointer on the display in response to the cursorcontrol signal.
 13. A method according to claim 11, wherein eachGUI-component being adapted to generate at least one respective outputcontrol signal upon a user manipulation of the component.
 14. A methodaccording to claim 11, further comprising producing at least a firstsignal of the non-cursor controlling event output signals based on adynamic development of the eye-tracking data signal.
 15. A methodaccording to claim 14, wherein the first signal representing aparticular gaze pattern over the display.
 16. A method according toclaim 15, wherein the at least one GUI-component being adapted tointerpret the first signal as an estimated intention of the user, andtrigger a user manipulation of the component in response to theestimated intention.
 17. A method according to claim 15, wherein atleast one GUI-component being adapted to interpret the first signal asan estimated attention level of the user, and trigger a usermanipulation of the component in response to the estimated attentionlevel.
 18. A method according to claim 15, wherein at least oneGUI-component being adapted to interpret the first signal as astate-of-mind parameter of the users, and trigger a user manipulation ofthe component in response to the state-of-mind parameter.
 19. A methodaccording to claim 11, further comprising: receiving at least oneauxiliary input signal, and producing the set of non-cursor controllingevent output signals on the further basis of the at least one auxiliaryinput signal.
 20. A method according to claim 11, wherein the at leastone auxiliary input signal being originated from at least one of abutton, a switch, a speech signal, a movement pattern of an inputmember, a gesture pattern, a facial expression and an EEG-signal.
 21. Acomputer program directly loadable into the internal memory of acomputer, comprising software for controlling the steps of claim 11 whensaid program is run on the computer.
 22. A computer readable medium,having a program recorded thereon, where the program is to make acomputer control the steps of claim 11.